Repository navigation
ci(oceanbase): bound startup and balance migration workers - #2515
Zxf-xufeng wants to merge 28 commits into
Conversation
hidb4ai
left a comment
There was a problem hiding this comment.
Merge readiness blocker: this head conflicts with current master (9697d5f264adfafd9f7b15f19e1ca6b0176ab8c6) in .github/workflows/e2e-harness.yml, at Maintain the registered Server chain on real OceanBase without skips. git merge-tree --write-tree d8492f56 9697d5f2 reproduces the content conflict: this PR adds the diagnostic wrapper with a 600-second deadline, while master extends that maintenance step to 900 seconds for fixture cleanup and JUnit output. Please reconcile the wrapper with the updated maintenance timeout and rerun the affected acceptance checks on the resolved commit. The current successful CI run does not validate that integration.
|
Addressed the merge-readiness review in 08a34940. Merged the latest fetched master ( Also fixed the diagnostic lifecycle: the native snapshot is collected before replay, and failed OceanBase replay collects a separate bounded snapshot before Local validation passed: |
|
|
||
| - name: Run clean migration evidence, grants and vectors without skips | ||
| # Up to 16 cases per shard at 300s each, plus collection and termination grace. | ||
| # Measured clean migration work is about 21m per worker; retain the 90m group ceiling. |
There was a problem hiding this comment.
[P2] 分片 3→2 之后,这一步的 90 分钟上限不再覆盖逐用例超时之和
同一个 diff 里 --test-timeout 300 保持不变,但每片用例数从 16 增到 24,最坏路径从 16 × (300 + 10s kill-after) = 82.7 分钟(小于 90 分钟上限)变成 24 × 310s = 124 分钟(大于 90 分钟上限)。新注释把推导从「16 个用例 × 300s」改成「实测约 21m per worker」,等于用均值替换了原来的最坏情况保证。
数字来自本 PR 新增的 .github/oceanbase-migration-durations.json:SUITES 三个文件共 47 个用例、实测合计 2543.7s(42.4 分钟),二分片后每片 24 个、平均 21.2 分钟——注释里的「21m」是准的,但它是最坏情况之外的另一种口径。
影响:本 PR 要治的失败模式正是 OceanBase 启动/迁移阶段的挂起。同片内多个用例挂起时,scripts/oceanbase_ci.py run 的 wait(timeout - min(30, timeout/2)) → collect(budget=20) → execution.json / before-timeout/ 这条取证路径还没轮到,步骤就先被 90 分钟上限杀掉,失败归因退化成「步骤超时」而不是带诊断的逐用例超时。同一个作业里其它三步仍然自洽(background-metrics:1 个用例 / 10 分钟;startup-snapshot:最多 2 个 × 310s + 180s = 13.3 分钟 / 15 分钟;long-query:4 个 × 310s + 180s = 23.7 分钟 / 30 分钟),只有这一处被打破。
建议二选一:
- 提高该步骤上限到 ≥ 130 分钟,与 24 × (300 + 10s) 对齐;或
- 把
--test-timeout降到 200s(24 × 210s = 84 分钟 ≤ 90 分钟),并同步改写注释里的推导。
如果确实有意只按实测均值设上限,请在注释里写明这一点,否则后续维护者会继续按「上限 ≥ 逐用例超时之和」读这组数字。
There was a problem hiding this comment.
Fixed in ad6b9b5b, with cold-start headroom completed in da11dcd4. The duration-based assignment produces 22 and 25 clean-migration cases. The larger shard needs up to 25 × (300 + 10) + (180 + 10) = 7,940s (132m 20s), including collection and termination grace. The step now allows 140 minutes. The job allows 250 minutes: 210 minutes of test-step budgets plus 40 minutes for a separately bounded cold image pull, startup, diagnostics and upload. The per-case deadline remains 300 seconds.
The regression check fails with the original 90-minute step and passes with the corrected budget. The runner still stops at the first failed case; the budget also covers slow passing cases before that failure. This changes ceilings, not the number of executions or retries. A separate cold-pull regression reproduces the previous 180-second startup failure and verifies warm-cache reuse and failure propagation.
The same changes are included in #2526. Both preceding runs passed all Python 3.11–3.14 unit/E2E jobs, but full backend CI is not yet green. Native logs from the separate background-metrics case show successive full-text index DDL still progressing at 277 seconds, before listener checks. That full-Server lifecycle now has a bounded 600-second deadline and a 15-minute step (including collection/grace); ordinary clean-migration cases retain 300 seconds. Lifecycle checkpoints are preserved, assertions remain intact, and no retry is added. This addresses premature termination of observed progress; the underlying engine latency and final pass still require CI verification. Final native acceptance is pending the new CI runs.
There was a problem hiding this comment.
The budget fix and latest master conflict resolution are pushed in 9636ae2. The larger clean-migration shard has 25 cases and a 360-second per-case ceiling. Including termination and collection, 25 × (360 + 10) + 190 = 9,440s fits the 160-minute step. The 13 long-query cases are assigned 6/7 and fit the 40-minute step. The 280-minute job covers 240 minutes of test-step ceilings plus 40 minutes for bounded setup, diagnostics and upload. No test retry or duplicate execution is added.
Both branches include master 7fef43d, pass make check, and combine locally without conflicts. Python jobs retain the 60-minute ceiling and cleanup reserve; master's website runner change is preserved. The resource producer has a checksum-preserving official Release API fallback for the observed HTTP 504 download failures.
The previous head passed all Python 3.11–3.14 unit/E2E jobs. Its downstream validation was incomplete when the conflict-resolution push superseded it. Run 42573624133 validates this new head; real-backend acceptance remains pending.
The same bounded, checksum-verified downloader now covers skill-up and pre-staged Windows NSIS tools. The plugin’s ordinary release URL reproduced HTTP 504 locally, while the official asset API returned bytes matching both the pinned SHA256 and Tauri’s upstream SHA1. This verifies the alternate artifact route; the native Windows installer and backend suites still require the linked CI run.
There was a problem hiding this comment.
[P2] long-query 步骤的 40 分钟上限同样只对当前基线成立,而该基线缺 5 个已收集的用例
clean-migration 这一条已核实(exact head c63f9b42,e2e-harness.yml / durations.json / test_oceanbase_ci.py 与 1742c5fb 逐 blob 相同):--test-timeout 360、步骤 160 分钟、作业 280 分钟;-k oceanbase --collect-only 收集 47 个节点,assigned_nodes 打包为 22/25,25 × (360+10) + 190 = 9,440s ≤ 9,600s;tests/test_oceanbase_ci.py 的预算用例在本机实跑 15 passed,把该步骤改回 90 分钟即失败(assert 5400 >= 9440)。这条视为闭合。
同一作业里还有一个步骤靠的是同一种等式,但没有对应的断言:
- 位置:
.github/workflows/e2e-harness.yml的Test long-query retrieval and admission without skips,注释写「Seven cases in the larger shard: 7 * (300s + 10s grace) + 190s collection < 40m」,timeout-minutes: 40,--test-timeout 300。 - 该步骤收集到 13 个 oceanbase 节点,但
.github/oceanbase-migration-durations.json的source_commit是4b6cf42a,早于引入这 5 个用例的提交,13 个里只有 8 个有时长:在册的是 27–163s(test_supported_long_query_preserves_all_terms_in_both_http_searches四条为 132/136/136/163s);缺失的 5 个全在tests/e2e/test_atomic_memory_fts_coverage.py——test_fts_coverage_counts_analyzer_terms_omitted_by_space_parser[oceanbase-{com,the,www}-legacy]、test_fts_coverage_keeps_distinct_terms_and_filters_before_limits[oceanbase]、test_hybrid_coverage_keeps_vectors_and_validates_rows_outside_lexical_matches[oceanbase]。 assigned_nodes对缺失节点用全体中位数(29s)估算,于是它们被排到最后、按「负载」而不是「条数」分摊。回复里的 13 个用例、6/7 正是这个估算下的结果。把缺失节点的耗时按同组在册用例的量级代入(≥120s)重跑同一算法,分配变成 8/5,最坏路径8 × (300+10) + 190 = 2,670s = 44.5 分钟 > 40 分钟;翻转阈值落在 100s 与 120s 之间。- 后果与原问题同形:同片多个用例挂起时,步骤上限先杀掉进程,
scripts/oceanbase_ci.py run的wait(timeout - min(30, timeout/2))→collect(budget=20)→execution.json/before-timeout/这条取证路径还没轮到,失败退化成「步骤超时」。
差别只在覆盖面:tests/test_oceanbase_ci.py 里 largest_shard × (case_timeout + 10) + 190 的断言只对 clean-migration 步骤生效(该用例名即 test_clean_migration_shard_budget_covers_slow_cases_and_keeps_cleanup_time),long-query、startup-snapshot、background-metrics 都没有同类断言,所以 master 再加用例时只有注释在维持等式。
建议:
- 用包含这 5 个节点的运行刷新
.github/oceanbase-migration-durations.json,并把source_commit前移到包含它们的提交; - 把该断言按步骤表驱动到所有运行
run_atomic_memory_clean_migration.py的步骤(从--test-timeout取逐用例上限、从该组SUITES的收集数取条数),而不是只钉住 clean-migration 一个步骤。
验证范围说明:13 个节点是用 -k oceanbase --collect-only 在本机 WSL 收集的(head 1742c5fb,上述 6 个文件在 c63f9b42 上 blob 相同);5 个缺失节点的真实耗时无法在无 OceanBase 的环境测得,因此这一条的依据是「基线缺测 + 打包会翻转」的算术,不是实测耗时。
Reuse the entrypoint fix from #2537 to unblock backend acceptance. Co-authored-by: knqiufan <knqiufan@foxmail.com>
hireview001
left a comment
There was a problem hiding this comment.
- CI failed: CI result (failure).
- CI failed: harness-validation / Atomic Memory clean migration (OceanBase) (failure).
- CI failed: harness-validation / Acceptance (oceanbase) (failure).
- CI failed: harness-validation / Recall effort (OceanBase) (failure).
- CI failed: harness-validation / Atomic Memory clean migration (OceanBase shard 1/2) (failure).
- CI failed: harness-validation / Atomic Memory clean migration (OceanBase shard 0/2) (failure).
- CI failed: desktop-validation / linux (failure).
- CI failed: desktop-validation / macos (macos-15) (failure).
- CI failed: desktop-validation / windows (failure).
- CI failed: desktop-validation / macos (macos-15-intel) (failure).
- Additional CI issues: PR checks.
…reliability # Conflicts: # .github/workflows/e2e-harness.yml # .github/workflows/master.yml
|
The conflicts with master 7fef43d are resolved and pushed in 9636ae2. I merged master independently into both PR branches, retained the Python test/cache budgets and cached license checker, and preserved master’s website runner change. GitHub now reports both #2515 and #2526 as MERGEABLE; the branches also combine locally without conflicts. Both pass |
…reliability # Conflicts: # .github/workflows/master.yml # tests/test_ci_pipeline.py
Which issue or RFC does this PR close?
Follow-up to #1945 and #2243. No issue is automatically closed.
Rationale for this change
OceanBase CI repeats general regressions, repeatedly initializes full Server schemas, and can terminate before collecting failure evidence or saving caches. This change reduces duplicate work and gives each bounded test command time to finish and preserve diagnostics.
What changes are included in this PR?
Are there any user-facing changes?
No public API or persisted-format changes. OpenDAL's projection behavior is preserved with a type-safe validation call. Other changes affect CI, diagnostics and test fixtures.
How was this change tested?
make check: passed after merging master7fef43d6.make opendal-testbootstrap hit a PyPI timeout. Workflow action pins and diff checks passed.AI usage statement
OpenAI Codex (GPT-6) assisted with analysis, implementation, regression tests and PR preparation.